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Abstract. Based on a review of the recent technical literature there is little question that a serious training gap exists for fifth- 
generation fighters, primarily arising from the need to provide their own red-air. There are several methods for reducing this gap, 
including injecting virtual and constructive threats into the live cockpit. This live- virtual-constructive (LVC) training approach 
provides a cost effective means for addressing training needs but faces several challenges. Technical challenges include data links 
and information assurance. A more serious challenge may be the human factors dimension of representing virtual and constructive 
entities in the cockpit while ensuring safety- of-flight. This also needs to happen without increasing pilot workload. This paper 
discusses the methods Rockwell Collins and the University of Iowa’s Operator Performance Lab use to assess pilot workload and 
training fidelity measures in an LVC training environment and the research we are conducting in safety^ of-flight requirements of 
integrated LVC symbology. 


1.0 INTRODUCTION 

In early 201 1 , RAND Corporation published 
a Project Air Force report titled "Investment 
Strategies for Improving Fifth Generation 
Fighter Training" that analyzed the current 
state of training for fifth generation fighters. 
Using a variety of data sources the authors 
concluded that a significant training gap 
exists. The report also stated that bringing 
training to acceptable levels would require 
significant investment by the services. 

One of the major factors indicating a training 
gap exists is the so-called “Insatiable 
Demand for Red Air”, or opposing forces. 
F-22 units are required to provide their own 
red-air component, meaning the few v. 
many training exercises utilize almost all 
available aircraft. To further complicate the 
matter, using similar aircraft as red air 
degrades the training received by blue air 
pilots. Other factors contributing to this 
conclusion are range and airspace 
limitations that result from increasing 
airspace requirements for commercial and 
general air traffic and the lack of 
documented training requirements. It is 
likely that these factors will only increase as 
the F-35 enters the operational inventory. 


These conclusions lead to the obvious 
question, “How do we close the training 
gap?” Five options were discussed in the 
RAND 201 1 report, and analysis of the 
costs associated with each option led the 
authors to conclude “in the long run, 
development of the Live-Virtual- 
Constructive (LVC) ability to inject simulated 
and constructive threats into live aircraft 
may be the only fiscally responsible 
approach to improving training” [1], 
Calculating the return on investment for 
developing this capability is complicated. 
The technology to accomplish this in not yet 
fully developed, and as such, the actual 
costs associated with it are unknown. 

Cutting funding for live training to enable 
significant investments in LVC is not the 
answer. In fact, this would only widen the 
training gap. Instead, smaller, strategic 
investments must be made to address the 
challenges and mature the technology 
before any significant investment is made. 

2.0 HUMAN FACTORS OF LVC 

In order to illustrate our perception of what 
LVC is, the challenges we associate with it, 
and our approach when performing 
research in this domain, we propose the 
taxonomy shown in Figure 1 . We have 
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proposed this taxonomy with the hope that 
others will add to or challenge its 
applicability. 


Domain 



Real 

World 

Virtual 

World 

Human 


V 


Machine 




Figure 1. Live, Virtual, Constructive and 
Autonomous as related to the worlds and 
executors of actions in those worlds 

The taxonomy illustrates that there are 
humans and computers executing actions in 
both the real and virtual worlds. As true 
LVCA (A represents autonomous systems) 
environments become a reality, the line 
between the real and virtual worlds will be 
blurred, requiring humans and computers to 
interact in a new, blended environment. 
Operating in this blended environment gives 
rise to safety challenges described further in 
the following sections. 


participants. The complete end-to-end 
connection of participants is at the crux of 
cracking the LVC challenge. If there are no 
meaningful ways to enable information 
exchange between the players, then 
discussion of the LVC topic is moot. The 
virtual and constructive threat information 
cannot be injected into the live platform if it 
cannot reach it. 

Unlike virtual platforms on the ground that 
can be easily connected via optical fiber, the 
live platform data links must go over the 
radio. The real world presents challenges 
for maintaining connectivity at long 
distances when the live participants move at 
various speeds, with different orientations 
with respect to receiving antennas, etc. The 
sheer volume of data required for real-world 
scenarios also presents a significant 
obstacle and drives the requirements for 
data link robustness, as well as traffic 
shaping and scheduling algorithms. It may 
not be appropriate or practical to burden 
tactical networks with training data, as is 
done in the approach discussed by Lechner 
and Wokurka [2], Appended LVC systems 
may require a separate data link, allowing 
preservation of tactical network bandwidth 
for operational communications rather than 
for training data. 


2.1 Challenges 

Whereas LVC can provide a cost effective 
means for addressing training needs, there 
are several challenges that must be 
overcome. Researchers active in this 
domain are quite familiar with two of the key 
technical challenges, data links and 
information assurance, but the impact on 
the operators has yet to be studied in 
sufficient detail. We will touch briefly on 
each of these technological challenges, and 
how they relate to the human dimension, 
and then discuss the human factors in more 
depth. 

2.1.1 It Is (Almost) All About 
Connectivity 

As the section title states, one of the biggest 
obstacles is the ability to connect 


Incomplete or inconsistent representation of 
virtual/constructive training data may be the 
result of data link latency, loss of data link, 
or data routing issues. Large entity counts 
that may be generated by LVC federations 
can quickly exceed the physical limits of the 
data link, giving rise to the potential for lost 
or non-transmitted tracks. Entities that are 
conveyed through a data link may generate 
tracks that can become stale or 
desynchronized when the data link breaks 
during aircraft maneuvering or due to flight 
outside the data link operational envelope. 
Tracks that become stale or disappear due 
to data link outages will cause confusion in 
LVC participants, especially in exercises 
with very large entity counts. The confusion 
of tracks among LVC exercise participants 
may result in degraded training and may 
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also cause unsafe merges between live 
aircraft and non-participating traffic or 
interlopers. 

2.1.2 If Data Can Reach the 
Destination. Shouldjt ? 

Assuming the data link challenge was 
solved and traffic could flow from one 
participant to another, it does not mean it is 
allowed to flow, according to classification 
rules. This cross-domain challenge is 
related to the fact that the virtual world must 
address security concerns often present in 
the real world scenarios. For example, an 
LVC system connecting a U.S. Marine Joint 
Terminal Attack Controller (JTAC) at the 
unclassified Camp Pendleton simulation 
facility to the Nevada Test and Training 
Range, which operates a number of Secret 
and Top Secret aircraft platforms, would be 
completely useless without a process for 
handling the hierarchies of data distribution 
and crossing classification domains. 



Figure 2. Cross domain solutions are a 
necessary component of network-centric 
environments, such as LVC systems, that 
require information sharing across all 
security enclaves 

Because High Level Architecture (HLA), the 
protocol used for many distributed 
simulations, is based on a publish-subscribe 
mechanism, any participant can subscribe 
to information from other participants. From 
a security perspective, this reduces the 
amount of control over the data flowing 
between the participants and presents the 
risk that sensitive information about planned 
missions, performance capabilities 
(platform, sensors, weapons, etc.), locations 
of facilities, task force composition, tactics, 


doctrines, etc. may be disclosed. Moller et 
al. [3] explore several different techniques 
for handling the cross domain challenge, 
most of which involve limiting and/or 
obfuscating the data being passed between 
the participants. 

What impact does this limited, degraded, or 
complete lack of information have on the 
trainees? Is the information good enough to 
still provide effective transfer of knowledge? 
Or, perhaps more concerning, does it result 
in negative training? We pose these 
questions knowing that there currently are 
no definitive answers. 

2.1.3 The Human Dimension 

Whereas the integration of virtual and 
constructive elements into live training 
opens up new avenues for training, it also 
raises concerns about safety of flight. 

When needed, aircrew integrated into this 
blended environment must be able to 
differentiate between actual and artificial 
entities or threats. Additionally, aircrew 
manning live aircraft will be expected to 
continue to execute their original duties 
while coping with with the changes 
introduced by these modifications. 

The first, and probably most obvious, 
concern is an increase in cognitive workload 
- the complexity and confusion that could 
occur when additional information is 
introduced into an already complex systems 
management problem. Minor increases in 
pilot workload can have an impact on safety 
critical tasks, such as visual search strategy 
and threat deconfliction methodology. The 
training benefits of LVC could be easily 
offset if this flight safety hazard is not 
mitigated appropriately. 

A second area of concern lies in the 
misinterpretation of virtual and/or 
constructive symbology or annunciations. 
Lacking appropriate safeguards, a pilot 
could misinterpret a live entity as a virtual 
input and, considering it to be a training 
target, perform inappropriately threatening 
maneuvers or obtain radar lock against a 
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live aircraft not participating in the training 
exercise. These types of actions could 
result in a real-world rules-of-engagement 
violation. Conversely, misinterpreting a 
virtual entity could lead to risks such as fuel 
emergencies as a result of flying to a 
simulated tanker, performing high risk 
maneuvers to engage or evade a virtual 
entity perceived to be a genuine threat, or 
disrupting the exercise due to a traffic 
conflict that does not exist in the real world. 

Until quite recently, these concerns caused 
decision makers to be reluctant to add LVC 
capabilities to operational and readiness 
training. In certain training environments, 
instructors can provide a margin of safety 
for an overloaded student, but when an LVC 
environment is applied to operational 
training and readiness there is no dedicated 
instructor. Mitigating the safety of flight 
concerns can possibly be achieved through 
the use of distinct symbology and/or 
annunciations that allow the pilot to quickly 
distinguish between information based on 
real-world data and that generated by virtual 
and constructive participants. However, any 
additional cognitive workload induced by 
blended symbologies, annunciations, and 
training scenarios must be handled in a 
manner that does not further compromise 
pilot safety. 

2.2 Quantifying the Human 
Dimension 

Until quite recently, the human dimension 
has been largely overlooked. Interest in 
quantifying the impacts on operators has 
increased, as evidenced by the Office of 
Naval Research (ONR) Broad Agency 
Announcement (BAA) 11-005 [4], initially 
posted on December 3 rd , 2010. The BAA 
contains three technical areas that focus on 
the training fidelity concerns in the live, 
virtual, and constructive domains 
respectively. 

Our research team has a four year history of 
conducting LVC exercises integrated in the 
U.S. airspace, and has developed a real- 
time workload measurement system for use 


in both aircraft and flight simulators. 
Although this neuroergonomic operator 
monitoring and evaluation system [5] [6] 
was developed independent from the LVC 
research conducted by the team, it is well 
suited for pilot performance assessment in 
integrated LVC environments. Our research 
team has a number of experimentation 
systems already in place, including two L-29 
jet aircraft and several reconfigurable 
simulators, which are equipped with these 
tools to evaluate pilot workload and training 
effectiveness. Successful tests have 
already been conducted in flight simulators, 
research vehicles, and flight test aircraft [7], 

The Cognitive Avionics Tool Set (CATS), 
shown in Figure 3, uses neurocognitive and 
physiological signals to generate a real-time 
measure of cognitive workload. CATS uses 
data from respiration, electroencephalogram 
(EEG), electrocardiogram (ECG), eye 
tracking, and galvanic skin response (GSR) 
sensors to measure a wide range of human 
neural and physiological characteristics. 
State-of-the art active shielding electrodes, 
combined with filtering and processing 
techniques, are used to reduce the effects 
of adverse noise and signal acquisition. 

The CATS framework was designed to 
easily accommodate new sensors and 
analysis techniques. 



Figure 3. CATS workload monitoring system 
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A highly integrated and rugged ized 
instrumentation package, with a single point 
umbilical connection to the aircraft or flight 
simulator, simplifies the deployment of 
neurocognitive and physiological sensors. 
EEG electrodes have been integrated into 
the liner of a flight helmet and the 
respiration belt and ECG electrodes are 
worn under the flight suit. All peripheral 
electronics have been integrated into a pilot 
survival vest. 

The Quality of Training Effectiveness 
Assessment (QTEA) tool [6] uses data from 
CATS and quantifies pilot performance 
against set performance requirements. 

Both systems record a rich dataset, 
containing hundreds of variables that 
constitute some form of measurement of 
effectiveness, for further offline analysis. 
CATS and QTEA provide powerful, state of 
the art analysis capabilities, and our current 
research plan involves applying these tools 
in an LVC environment to assess 
symbologies for injecting virtual and 
constructive entities into live platforms. 


3.0 DISCUSSION 

Workload and performance are fundamental 
considerations when designing new aircraft 
or weapons systems. To successfully 
augment real world inputs with training 
inputs coming from simulation systems the 
original design factors must be modified, 
and the viability of these modifications must 
be tested in the cockpit. Using both in-flight 
and ground-based evaluation systems, our 
team is conducting research into LVC- 
aware avionics symbology and annunciation 
sets and guidelines for LVC-aware avionics. 

Candidate symbology design and 
development of simulation based prototype 
displays is being driven by requirements 
definition and stakeholder needs. Human- 
in-the-loop testing will gather data on the 
impact to safety of flight and training 
effectiveness of the symbologies. Tests will 
be conducted in an environment drawn from 
an application of mission tasks to a notional 


LVC training scenario, and performance 
data will be gather by CATS and QTEA. 

The tools will be deployed within our 
existing LVC framework to obtain real-time 
measures of workload, situation awareness, 
and pilot performance on the basis of 
neurocognitive and physiological measures 
as well as mission and flight technical 
performance measures. The combined 
system of human performance assessment 
tools and LVC capabilities available to our 
team provides a unique evaluation platform 
for collecting quantitative data on the 
suitability, workload, and cognitive demands 
of the proposed symbologies and 
annunciations. 

Our experimentation system, shown in 
Figure 4, includes two L-29 jet aircraft with 
modified evaluation cockpits in the rear 
seat, integrated instrumentation pods, a 
ground support infrastructure, a 
reconfigurable single-seat simulator and an 
operator monitoring and evaluation system. 
This constellation of assets will allow our 
team to test the performance of multiple 
crews in an LVC exercise and conduct 
human-in-the-loop testing in both the 
reconfigurable flight simulators and LVC- 
enabled live aircraft. The system will also 
allow evaluation of the display prototypes to 
make strong claims about the validity and 
effectiveness of various virtual and 
constructive symbology insertion methods in 
a much more effective manner than 
traditional human factors data collection 
techniques, such as pilot surveys. 

At this stage in our research, we foresee 
symbology variations using multiple 
stimulus dimensions for differentiating 
between live, virtual, and constructive 
entities. Variations include, but may not be 
limited to, symbol color, size, fill, shape, text 
legend, style (dashed, dotted, solid), 
blinking (speed, duty cycle), fading, and 
contrast polarity. Variations in auditory 
annunciations to distinguish between entity 
types will also be investigated, including 
dimensions such as voice (male vs. female), 
amplitude, frequency, waveform, envelope 
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(attack, sustain, release, decay), and stereo considered are type of display (visual, 

vs. mono. Additional variables that will be auditory), display location (head-up, head- 



c. L-29 Front Cockpit d. L-29 Rear Cockpit 

Figure 4. Instrumented Flight Simulator and Matching L-29 Flight Test Aircraft 


down, helmet), and scenario related 
settings. We are also considering the 
potential negative impacts of allowing 
trainees to distinguish between live, virtual, 
and constructive entities, and the possibility 
that the highest quality of training may occur 
when they cannot. 

As a byproduct of the design and testing 
efforts we plan to deliver LVC training safety 


and training effectiveness assessment 
methods and preliminary design guidelines 
for LVC datalink technology. Where 
practical we will develop detectable criteria 
to indicate a potential loss of real-world 
situation awareness or other safety of flight 
concerns, and propose display modes 
and/or annunciation to prevent the 
development of flight risks that may arise 
when virtual and constructive entities are 
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injected into live systems. Our goal is to 
provide traceability from measured pilot 
performance, through analysis and design, 
back to the mission tasks and requirements 
definition. We will show how specific 
symbologies and avionics configurations 
map to the ability to safely conduct LVC 
training to meet training and readiness 
requirements. Upon completion of this 
research we will demonstrate that the final 
design meets the requirements for mission 
training, and has a known, negligible impact 
on safety of flight. 


4.0 CONCLUSION 

A significant training gap exists for fifth- 
generation fighters, and a recent study 
identified the LVC ability to inject virtual and 
constructive entities into the live aircraft as 
the most cost effective solution. Although 
the technical challenges associated with 
integrating live equipment into LVC 
environments has received significant 
attention, the effect on the operators has not 
been studied in detail. This human 
dimension can have a significant impact on 
the training benefit and acceptance of LVC 
for operational and readiness training. 

Our research team is conducting design and 
experimentation work to determine LVC- 
specific guidelines and symbologies that 
enable safe and effective insertion of virtual 
and constructive entities into live systems. 
An iterative research cycle that draws upon 
the team’s extensive human factors 
experience will continue to evaluate and 
improve symbology and update guidelines 
throughout the research period. Because of 
the comprehensive instrumentation we have 
available, and our successful testing track 
record, we will be able to identify design 
considerations and present strong 
quantitative and qualitative evidence to 
support our final conclusions. At the time 
this paper was written, preliminary results 
were not yet available. 
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